Peered proctoring

ABSTRACT

Systems and methods described herein utilize a distributed server network to allow for the real-time distribution of copies of a data stream uploaded from a computing device. The uploaded data stream corresponds to an environment surrounding a user (exam taker) of the computing device during execution of a testing routine for an examination. Providing copies of the data stream in real-time allows proctors as well as other test assessment authorities to ‘peek in on the exam taker&#39;s environment during the exam. As a result, the environment surrounding the exam taker can be monitored in real-time to determine whether any visual or audio activity in the environment constitutes activity not in accordance with the exam protocol. Following an affirmative determination that the exam taker violated an exam taking protocol, a proctor or other test assessment authority can then make a final or early decision regarding disciplinary action.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation and claims the prioritybenefit of U.S. patent application Ser. No. 14/179,410 filed Feb. 12,2014 and set to issue as U.S. Pat. No. 9,092,991 on Jul. 28, 2015, whichis a continuation and claims the priority of benefit of U.S. patentapplication Ser. No. 12/913,694 filed Oct. 27, 2010, now U.S. Pat. No.8,713,130 issued on Apr. 29, 2014, which is a continuation-in-part andclaims the priority benefit of U.S. patent application Ser. No.12/850,136 filed Aug. 4, 2010, the disclosures of all of which areincorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to streaming multimedia data.More specifically, the present invention concerns managing the real-timedistribution of a live data stream such as a live video data stream froma computing device.

2. Description of the Related Art

Examinations are used to determine the ability of an exam taker such asa student or prospective practitioner as it pertains to proficiency in aparticular subject or skill set. For example, a student might take anexam to determine whether the student possesses requisite knowledge in aparticular subject that might be related to receiving a degree orcertificate. A prospective practitioner of law or medicine similarlymight sit for an examination to determine their competence as itpertains to practicing in that profession.

Students or prospective practitioners have historically gathered at thedesignated locale for an examination on a proscribed date and time.Examination materials are then handed out by a testing authority and theexam begins. During the allotted time, the exam takers read questionsand provide answers on a provided answer sheet or in a ‘blue book.’Throughout the course of examination, a teacher or proctor keeps carefulwatch over the exam takers to ensure that no instances of cheating aretaking place. While a single proctor may be able to observe a smallgroup of exam takers, such observation becomes more difficult for alarger exam taking pool or for a group of exam takers utilizing laptopcomputers or other computing devices.

The increased popularity of distance learning has also complicatedproctoring of examinations. The distance learning instructional modeldelivers education material and information to students who are notphysically on ‘site’ at an education facility. Distance learningprovides access to learning opportunities when the source of theinformation and the student are separated by time or distance if notboth. Thousands of distance learners may be involved in a particulardistance learning program or course at any given time.

Distance learning is no different than any other educational program inthat there is a need to verify the qualifications of students throughexamination. Because distance learners are not collectively gathered ata physical learning institution such as a university, the distancelearning program often requires that the students attend an examinationcenter—which defeats a purpose of distance learning—or administers anexamination online. An online examination is difficult to proctor as anexam taker could be taking an examination in one window of a web browserwhile looking up answers in another window via the Internet. An examtaker could also utilize a ‘chat’ or ‘messaging’ application to relayquestions to and receive answers from a knowledgeable third-party. Thevalue of online examinations is, therefore, questionable and calls intoquestion the overall value of the corresponding class or degree program.

Techniques for remotely monitoring the visual and/or audio environmentsurrounding the exam taker, through the use of an environment capturedevice such as a video camera or other capture device, are disclosed incommonly-owned U.S. patent application Ser. No. 12/723,666, thedisclosure of which is incorporated herein by reference. In order toensure the integrity of the exam process, the real-time audio and/orvideo data streams of the testing environment can be provided over anetwork to a remote proctor who is responsible for remotely observingthe administration of the exam.

Uploading the data stream and subsequently providing it in real-time toa remote proctor is challenging for a number of reasons including highbit rates, delay, and loss sensitivity. In particular, the data streamis vulnerable to upload restrictions and/or disruptions along the ‘firstmile’ from the test taker's computing device into the network, which cancause packet loss and delay of data in the data stream. This can be dueto physical upload bandwidth limitations through the use of dial-upmodems, or other upload restrictions such as network service providerlimitations. As a result, the so-called ‘first mile’ can become abottleneck, which can render further downstream techniques ineffective,since these downstream techniques cannot improve the bit rate and streamintegrity of the initially uploaded data stream.

Similarly, the data stream is vulnerable to download restrictions and/ordisruptions along the ‘last mile’ from the network to the remoteproctor's computing device. As a result, the so-called ‘last mile’ canalso become a bottleneck, which can render the previous upstreamtechniques for improving the bit rate and stream integrity ineffective.These download restrictions and disruptions also hamper the observationof the exam taker during the administration of the exam, which may callinto question the integrity of the exam process.

There is a need in the art for managing the distribution of a datastream, which addresses first mile and last mile vulnerability issues.There is a similar need for improved remote proctoring of large scaleexaminations such that a small number of proctors can properly maintainthe integrity of the testing environment, notwithstanding the largenumber of exam takers.

SUMMARY OF THE CLAIMED INVENTION

Embodiments of technologies for managing distribution of a data streamfrom a distributed computing device network are claimed.

In a first claimed embodiment, a computer-implemented method formanaging distribution of a data stream from a distributed computingdevice network includes receiving a data stream into the distributedcomputing device network. The data stream corresponds to an environmentsurrounding a user of a first remote computing device during executionof a testing routine for an examination. The method includes receiving arequest to transmit the data stream to a second remote computing device.The request identifies a geolocation of the second remote computingdevice and is received over the network during execution of the testingroutine. A computing device of the distributed computing device networkis selected to process the request by transmitting the data stream tothe second remote computing device. The computing device is selectedbased on the geolocation of the second remote computing device. Themethod also includes transmitting the data stream from the selectedcomputing device in the distributed computing device network to thesecond remote computing device in response to the request.

In a second claimed embodiment, a non-transitory computer-readablestorage medium has a program embodied thereon. The program is executableby a processor of a computing device to perform a method for managingdistribution of a data stream from a distributed computing devicenetwork. The method includes receiving a data stream into thedistributed computing device network. The data stream corresponds to anenvironment surrounding a user of a first remote computing device duringexecution of a testing routine for an examination. The method includesreceiving a request to transmit the data stream to a second remotecomputing device. The request identifies a geolocation of the secondremote computing device and is received over the network duringexecution of the testing routine. A computing device of the distributedcomputing device network is selected to process the request bytransmitting the data stream to the second remote computing device. Thecomputing device is selected based on the geolocation of the secondremote computing device. The method also includes transmitting the datastream from the selected computing device in the distributed computingdevice network to the second remote computing device in response to therequest.

In a third claimed embodiment, a system for managing distribution of adata stream from a distributed computer network includes a plurality ofcomputing devices forming a distributed computer network. Each of theplurality of computing devices includes a processor, memory, and anetwork interface. A selected computing device of the plurality ofcomputing devices receives at the network interface a data stream intothe distributed computing device network. The data stream corresponds toan environment surrounding a user of a first remote computing deviceduring execution of a testing routine for an examination. The selectedcomputing device also receives at the network interface a request totransmit the data stream to a second remote computing device. Therequest identifies a geolocation of the second remote computing deviceand is received over the network during execution of the testingroutine. The selected computing device executes instructions stored inthe memory. Upon executing the instructions, the selected computingdevice selects a computing device of the distributed computing devicenetwork to process the request by transmitting the data stream to thesecond remote computing device. The computing device is selected basedon the geolocation of the second remote computing device. Upon executingthe instructions, the selected computing device also transmits the datastream from the selected computing device in the distributed computingdevice network to the second remote computing device in response to therequest.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a system for providing optimization of entry andsubsequent real-time distribution of a data stream uploaded into adistributed server network.

FIG. 2 illustrates a method for implementing an online proctoredexamination.

FIG. 3 illustrates a method for uploading a data stream of theexamination environment surrounding the exam taker during an onlineproctored examination.

FIG. 4 is illustrates a method for selecting an entry server to receivean encoded data stream.

FIG. 5 illustrates a method for transmitting an uploaded data stream ofan examination environment to a proctor or other third party assessmentauthority during an online proctored examination.

FIG. 6 illustrates a first interface that may be utilized in proctoringan online examination.

FIG. 7 illustrates a second interface that may be utilized in proctoringan online examination upon detecting aberrant behavior in the interfaceof FIG. 6.

DETAILED DESCRIPTION

Systems and methods described herein utilize a distributed servernetwork to allow for the real-time distribution of copies of a datastream uploaded from a computing device. The uploaded data streamcorresponds to an environment surrounding a user (exam taker) of thecomputing device during execution of a testing routine for anexamination. A video camera device or other visual and/or audioenvironment capture device may be used to generate the data stream. Thisdata stream is encoded and uploaded into the distributed server network.The distributed server network can then transmit copies of the datastream for playback on the computing device of a remote proctor, as wellas other third party test assessment authorities (e.g., a professor),during the administration of the examination.

Providing copies of the data stream in real-time allows proctors as wellas other third party test assessment authorities to ‘peek in’ on theexam taker's environment during the exam. As a result, the environmentsurrounding the exam taker can be monitored in real-time to determinewhether any visual or audio activity in the environment constitutesactivity not in accordance with the exam protocol. Following anaffirmative determination that the exam taker violated an exam takingprotocol, the proctor or other third party test assessment authority canthen make a final or early decision regarding disciplinary action.

The management of the distribution of the copies utilizing thedistributed server network also allows for the real-time sharing andshifting of proctoring responsibilities, as well as those of the thirdparty test assessment authorities. Advantageously, this provides forremote proctoring of large scale examinations such that a small numberof proctors and third party authorities can properly maintain theintegrity of the testing environment, notwithstanding the large numberof exam takers.

FIG. 1 illustrates a system 100 for providing optimization of entry andsubsequent real-time distribution of a data stream uploaded into adistributed server network such as a content delivery network (CDN) 162.The system 100 of FIG. 1 includes computing devices 110, 112, 114 thatmay be utilized by a user (exam taker) to take an examination, CDN 162for administering an examination and for routing exam data includingreal-time data streams via a communications network 150, a centraloffice proctoring center 180, and geographically distributed proctoringcenters 190, 192, 194.

The computing devices 110, 112, 114 may be any sort of computing deviceas is known in the art. The computing devices 110, 112, 114 includememory for storage of data and software applications, a processor foraccessing data and executing applications, and input and output devicesthat allow for user interaction. The computing devices 110, 112, 114further include components that facilitate communication over thecommunications network 150 such as an RJ-45 connection for use intwisted pair based 10baseT networks or a wireless network interface cardallowing for connection to a radio-based communication network (e.g., an802.11 wireless network).

The computing devices 110, 112, 114 may each be a general purposecomputing device such as a desktop or laptop computer. The computingdevices 110, 112, 114 may belong to a particular user rather than beinga computing device dedicated to exam taking as might be found in anexamination center. Thin client or netbook client devices may beimplemented in the context of computing devices 110, 112, 114 as mightmobile computing devices such as smart phones.

Communication network 150 may be a local, proprietary network (e.g., anintranet) and/or may be a part of a larger wide-area network. Thecommunication network 150 may be a local area network (LAN), which maybe communicatively coupled to a wide area network (WAN) such as theInternet. The Internet is a broad network of interconnected computersand servers allowing for the transmission and exchange of InternetProtocol (IP) data between users connected through a network serviceprovider. Examples of network service providers are the public switchedtelephone network, a cable service provider, a provider of digitalsubscriber line (DSL) services, or a satellite service provider.Communication network 150 allows for communication between the variouscomponents of system 100.

The computing device 110 is representative of the computing devices 110,112, 114. As described below, a video camera device 120 or other visualand/or audio environment capture device is used to monitor the examtaking environment surrounding the user (exam taker) of the computingdevice 110 during administration of an examination. The video cameradevice 120 generates a real-time data stream of the exam takingenvironment. This data stream is encoded and uploaded by the computingdevice 110 into the CDN 162, which in turn transmits the data stream inreal-time to the central office proctoring center 180, and/or one ormore geographically distributed proctoring centers 190, 192, 194.

A proctor or other third party assessment authority may then view thedata stream via a streaming technique in real-time (i.e., duringadministration of the examination) utilizing a viewing application(e.g., view application 196), and determine if any visual activity inthe examination environment constitutes activity not in accordance withthe exam protocol. The term “streaming” as used herein refers to thetransmission of data in the data stream by the CDN 162 to the computingdevice of an end user such as proctor, without requiring that the datastream first be uploaded in its entirety to the CDN 162.

The CDN 162 is coupled to the communications network 150. The CDNincludes a CDN data center 170 and a plurality of entry servers 160-1 to160-6. In the system 100 illustrated in FIG. 1, six entry servers 160-1to 160-6 are shown. Notwithstanding the six illustrated servers, thesystem 100 may include more of less entry servers.

The CDN data center 170 also manages the real-time upload anddistribution of data streams of the examination environments from thecomputing devices 110, 112, 114, as well as the storing of the datastreams for retrieval and playback in a non-streaming fashion. The CDNdata center 170 includes an application server 172, database server 174,and media server 176. The application server 172, database server 174,and media server 176 are each a computing device and include memory, aprocessor for accessing data and executing applications, and componentsto facilitate communication over communications network 150. Theapplication server 172 includes an upload/download optimization module173 stored in memory and executed by a processor to invoke itscorresponding functionality.

The optimization module 173 is executable to handle assignment requestsfrom the computing devices 110, 112, 114 to upload data streams such asthose captured by the environment capture devices during administrationof an exam. These assignment requests are analyzed by the optimizationmodule 173 to select one or more “optimum” entry servers in theplurality of entry servers 160-1 to 160-6. The “optimum” entry server orservers then serve as an entry point or points for the data stream intothe CDN 162. The database server 174 stores entry server informationsuch as data indicating the physical or network location of each of theentry servers 160-1 to 160-6. This information can then be used by theoptimization module 173 to select the entry point or points for the datastream.

The optimization module 173 is also executable to handle requests toprovide the data stream to the central office monitoring center 180, anddistributed proctor monitoring centers 190, 192, 194. These requests areanalyzed by the optimization module 173 to select one or more “optimum”entry servers in plurality of entry servers 160-1 to 160-6. The“optimum” entry server or servers then serve as the source or sources ofthe data stream out of the CDN 162.

The optimization module 173 in the illustrated embodiment includes arate allocation algorithm and a network/location proximity algorithm.The rate allocation algorithm is used to determine the sending rate orupload bandwidth from each the computing devices 110, 112, 114 to assistin minimizing packet loss in the uploaded data streams. Similarly, therate algorithm is used to determine the receiving rate or downloadbandwidth to each of the central office monitoring center 180 anddistributed proctor monitoring centers 190, 192, 194 to assist inminimizing packet loss in the downloaded data streams.

The network/location proximity algorithm is used to determine thelocation of the computing devices 110, 112, 114 to assist in minimizingthe probability of packets arriving late to the selected entry server.Similarly, the network/location proximity algorithm is used to determinethe location of the distributed proctor monitoring centers 190, 192, 194to assist in minimizing the probability of packets arriving there late.The optimization module 173 is described in further detail below withrespect to FIGS. 3 to 5.

The application server 172 also operates as an origin server for theexam content during the administration and proctoring of an examination.The exam content includes the uploaded data streams.

The exam content may also include exam data such as the actual exam(e.g. prompts and questions) or other data concerning an exam, as welland answers to the questions provided by the exam takers during anexamination. Registration information of the exam takers, such as a nameor examination identification number as well as a password, may also bestored in the database server 174. Other registration information mightinclude a geographic location or address provided by the exam taker andproctor via a graphical user interface.

Biometric information such as a visual image of the exam taker may alsobe stored in the database server 174 and compared against a previouslystored and known ‘good’ image of the exam taker. A similar comparisonmay be made with respect to a voice print. Retinal scans and fingerprints, subject to the presence of the appropriate peripheral device,may also be stored and used for verifying exam taker identity. Theseperipheral devices may be implemented in the context of the use of videocamera device 120, microphone 130, or other environment capture device.

The media server 176 handles the requests for the non-real-time playbackof the uploaded data streams, which may be stored in the database server174 as well as the entry servers 160-1 to 160-6.

The entry servers 160-1 to 160-6, in conjunction with the applicationserver 172 and database server 174, provide high-performance upload anddelivery of the exam content to the exam takers and the proctors.

Each of the entry servers 160-1 to 160-6 is a computing device whichincludes memory, a processor for accessing data and executingapplications, and components to facilitate communication overcommunications network 150. The application server 172 handles thedistribution of copies of the exam content to the entry servers 160-1 to160-6. The exam content can then be stored and transmitted to the examtakers and proctors by way of the entry servers 160-1 to 160-6 using avariety of different load balancing techniques, instead of beingtransmitted directly by the application server 172. The distribution ofthe exam content to the entry servers 160-1 to 160-6 may includeon-demand or push-based mechanisms that move the exam content from thedatabase server 174 to the entry servers 160-1 to 160-6. In addition,the distribution of the uploaded data streams may consist of on-demandor push-based mechanisms that move the uploaded data streams betweenentry servers 160-1 to 160-6, without requiring that the uploaded datastream first enter the CDN data center 170.

The entry servers 160-1 to 160-6 are arranged at various physicallyremote locations or regions. For example, entry server 160-1 may belocated in North America, entry server 160-2 may be located in Asia, andentry server 160-3 may be located in Europe. The entry servers 160-1 to160-6 may be arranged at the “edge” of the communication network 150 sothat the entry servers 160-1 to 160-6 can be located physically close tothe exam takers and proctors. Due to this physical proximity, the entryservers 160-1 to 160-6 can provide faster, higher quality datatransmissions to achieve better load balancing, lower latency, andhigher throughput in delivering the data streams than may be otherwiseachieved using solely the application server 172

Central office proctoring center 180 is an operations center withcomputing devices staffed with one or more proctors or other testassessment authorities observing the data streams of various examinationenvironments for exam takers at one or more examination sites. Theseexamination sites may be physically remote from the central officeproctoring center 180. Examination sites can be examination centersdedicated to the offering of examinations, traditional classroomsettings, as well as personal space such as a home or office workspace.The geographically distributed proctoring centers 190, 192, 194 includecomputing devices which may belong to a particular proctor or other testassessment authority, rather than being a computing device dedicated toproctoring as might otherwise be found in the central office proctoringcenter 180. The proctors and other test assessment authorities at thecentral office proctoring center 180 and geographically distributedproctoring centers 190, 192, 194 may observe and analyze a variety ofdifferent types of information to help ensure the integrity of theexamination. The observation and analysis of information is described infurther detail below with respect to secure testing application 140 andvideo camera device 120.

The computing devices 110, 112, 114 are secured in order to preventaccess to files or other types of data such as notes, outlines, and exampreparation material during an examination, as well as preventing accessto applications that themselves allow for access to data. The computingdevices 110, 112, 114 may be secured for the taking of an exam asdescribed in commonly-owned U.S. patent application Ser. No. 12/571,666,the disclosure of which has been previously incorporated by reference.

The computing device 110, 112, 114 may be secured through the downloadand subsequent installation of a secure testing application (e.g.,secure testing application 140 installed on computing device 110).Secure testing application 140 may be downloaded from application server172 or another computing device coupled to communications network 150.Secure testing application 140 may also be installed from acomputer-readable storage device such as a CD-ROM. The secure testingapplication 140 may then be stored in memory at the correspondingcomputing device 110 and executed by a processor to invoke itscorresponding functionality.

Secure testing application 140 is a security application software thatprevents computing device 110 from accessing certain data orapplications that might otherwise be in violation of examinationregulations or protocols as identified by application server 172. Thesecurity application software 140 causes the computing device 110 tooperate in a secure mode by introducing certain changes to the systemregistry such that only those applications or files deemed necessary orappropriate by the exam administrator and as embodied in a correspondingexam protocol may be allocated address space, loaded into memory andultimately executed by the computing device 110.

For example, an exam protocol for a particular examination may denyaccess to a web browser, e-mail client, and chat applications such thatan exam taker may not electronically communicate with other individualsduring the examination. This particular protocol may be downloaded tothe computing devices 110 from the CDN 162 along with exam data. Thesecure testing application 140 then operates in accordance with thedownloaded testing protocol such that certain applications are notallowed to be loaded and executed.

Similar prohibitions or permissions may apply to hardware components ofthe computing device 110 as well as any number of hardware peripheralsthat might be introduced to the computing devices 110. Examples of suchperipherals that might be introduced include a second computer monitor,docking stations, a traditional full-sized keyboard as might be usedwith a laptop computer. Other peripherals might include thumb drives,‘time-shift’ recording devices that offer TiVo®-like functionality, aswell as any number of other plug-and-play peripherals.

The secure testing application 140 may also operate in conjunction withthe CDN 162 to properly execute an exam routine for the givenexamination event. For example, the exam routine may allow for the userto have access to all questions at any given time such that the user mayanswer and not answer questions at their leisure and subsequently returnto any questions at a later time for further review. The exam routinemay alternatively require the exam taker to lock in an answer or set ofanswers and have the same reported to the CDN 162 prior to receiving asubsequent question.

The secure testing application 140 may also observe activity on thecomputing device 110 during administration of an examination. If an examtaker attempts to make changes to the system registry that wereimplemented by the secure testing application 140, the secure testingapplication 140 may identify and report these attempts to the centraloffice monitoring center 180, and distributed proctor monitoring centers190, 192, 194.

The secure testing application 140 operates in conjunction with videocamera device 120 or other visual and/or audio capture device to monitorthe exam environment surrounding the exam taker of the computing device110. Video camera device 120, which may be a commercially available webcamera or other image acquisition device, generates a real-time datastream of the exam environment. If the exam taker leaves their seat oranother individual enters the exam area during the course of theexamination, the video camera device 120 will capture this visualinformation. The video camera device 120 provides the data stream to thesecure testing application 140. Other external devices may also be usedto gather environment data that can be provided in the encoded datastream uploaded to the CDN 172, such as a microphone 130 or otherenvironment capture device.

As described in more detail below, the secure testing application 140encodes and uploads the captured data stream into the CDN 162. The CDN162 in turn delivers the data stream to the central office monitoringcenter 180, and/or one or more geographically distributed proctormonitoring centers 190, 192, 194, during the administration of theexamination.

The computing device 191 at the proctoring center 192 is representativeof computing devices in the central office proctoring center 180 andgeographically distributed proctoring centers 190, 192, 194. Thecomputing device 191 may be any sort of computing device as is known inthe art. The computing device 191 include memory for storage of data andsoftware applications, a processor for accessing data and executingapplications, and input and output devices that allow for userinteraction. The computing device 191 further includes components thatfacilitate communication over the communications network 150.

The computing device 191 may be a general purpose computing device suchas a desktop or laptop computer. Thin client or netbook client devicesmay be implemented in the context of computing device 191 as mightmobile computing devices such as smart phones.

The computing device 191 includes a viewing application 196 stored inmemory and executed by a processor to invoke its correspondingfunctionality. The viewing application 196 operates in conjunction withthe CDN 162 to request, receive and display real-time data streams ofthe examination environment surrounding exam takers. The viewingapplication 196 is executable to request that the CDN 162 transmit datastreams to the computing device 191, as well as to request that the CDN162 transmit data streams to other proctor's computing devices.

The viewing application 196 may for example be a commercially availableapplication such as a web browser, or any other software applicationwhich can be utilized for proctoring an online examination as describedherein. The viewing application 196 may alternatively be created by thetesting administrator or otherwise developed specifically for proctoringof the online exam.

Similar to the discussion above with regards to the secure testingapplication 140, in some embodiments the viewing application 196 maycause the computing device 191 to operate in a secure mode.

A proctor or other test assessment authority at the central officemonitoring center 180 and/or the proctor monitoring centers 190, 192,194 may then view the data stream provided by the CDN 162 and determineif any visual activity constitutes activity not in accordance with theexam protocol. The proctor may then log the information for furtherassessment by the actual exam administrator (e.g., the professor orprofessional association administrating the examination) or make adirect inquiry of the exam taker as to the nature of the observedbehavior, provide a warning as to terminate that behavior, or otherwiseintervene in the testing routine to resolve the aberrant behavior.

The proctor or other test assessment authority viewing the data streammay also request that the CDN 162 temporarily or permanently transmitthe data stream to another computing device, such as one or more ofcentral office monitoring center 180 and the remote proctor monitoringcenters 190, 192, 194. This allows for real-time sharing and shifting ofresponsibilities among the proctors and the third party test assessmentauthorities. As a result, a small number of proctors and assessmentauthorities can properly maintain the integrity of the testingenvironment, notwithstanding the large number of exam takers.

As an example, a proctor may be responsible for simultaneous observationof a number of data streams of the examination environments surroundingdifferent exam takers. Upon detection of possible aberrant behavior by aparticular exam taker, the proctor may request that the data streams ofthe other exam takers currently be received also be transmitted to thecomputing device of one or more proctors. In doing so, the proctor canmore closely analyze the possible aberrant behavior by the particularexam taker, without allowing the remaining exam takers to go unobserved.This request may be generated automatically based on user input to theviewing application 196, such as the selection by the proctor that thedata stream be ‘exploded’ into a larger view (like in FIG. 7).

As another example, a proctor may request that the data stream of a testtaker exhibiting possible aberrant behavior be transmitted to thecomputing device of another proctor or third party test assessmentauthority (e.g., a professor) for the purpose of validating or making anaffirmative determination that the exam taker in fact violated an examtaking protocol. Following a determination that an exam taker violatedan exam taking protocol, the proctor or other third party testassessment authority can then make a final or early decision regardingdisciplinary action.

FIG. 2 illustrates a method for implementing an online proctoredexamination. In step 210, an account is created by an exam taker. Instep 220, an exam taker registers for and/or schedules an examination.In step 230, an exam taker engages in biometric enrollment andauthentication. In step 240 the exam is delivered and proctoringcommences at step 250. The aforementioned steps may be carried out asdescribed in co-pending U.S. patent application Ser. No. 12/723,667entitled “Secure Online Testing,” the disclosure of which isincorporated herein by reference.

Proctoring step 250 takes place over the course of the examination andmay be implemented as part of step 240. The proctoring step 250 utilizesa variety of security technologies and processes to deter and detectaberrance during the exam process. In particular, the exam takingenvironment surrounding the exam taker is monitored in real-time by aproctor utilizing a live video feed of the exam taker. The live videofeed is captured utilizing a video camera device or other visual and/oraudio environment capture device, and uploaded and distributed inreal-time by the CDN 162 utilizing the techniques described herein.

FIG. 3 illustrates a method 300 for uploading a data stream of theexamination environment surrounding the exam taker during an onlineproctored examination. At step 305, the computing device 110 transmitsan assignment request to the application server 172. The assignmentrequest is a request for identification information of an entry serverof the entry servers 160-1 to 160-6, which will serve as the entry pointfor the data stream into the CDN 162. The transmission of the assignmentrequest may be initiated automatically by the secure testing application140 upon successful completion of the enrollment and authentication ofthe exam taker. The assignment request will be described in more detailwith respect to FIG. 4.

At step 310, the application server 172 receives the assignment request.At step 315 the assignment request is analyzed to select an entry serverof the entry servers 160-1 to 160-6, which will be entry point for thedata stream. This determination is made based on one or more estimatedperformance metrics for a network path from the computing device 110 tothe selected entry server of the entry servers 160-1 to 160-6. Theperformance metrics can be computed using an estimated physical locationof the computing device 110, as well as entry server information such asthe physical location of each of the entry servers 160-1 to 160-6 storedin the database server 174. The analysis of the assignment request andthe selection of the particular entry server in step 315 are describedin further detail below with respect to FIG. 4.

At step 320, the application server 172 transmits assignment data to thecomputing device 110 requesting that the computing device 110 upload thedata stream to the selected entry server. The assignment data mayinclude data representing an IP address of the selected entry server, orother unique identifying information of the selected entry server whichcan be used to facilitate connection between the computing device 110and the selected entry server.

At step 325, the computing device 110 receives the assignment data. Atstep 330, the secure testing application 140 launches the examination onthe computing device 110. The secure testing application 140 may thenprovide the exam taker with instructions concerning the positioning ofthe video camera device 120 or other visual and/or audio environmentcapture device. Alternatively, this process may be undertaken at theenrollment and authentication stage.

The video camera device 120 operates in conjunction with the securetesting application 140 to generate a real-time data stream of theexamination environment. This data stream is provided to secure testingapplication 140, which in turn encodes the data stream into an encodeddata stream. The data stream is encoded in a format dependent upon theapplication, and in the illustrated embodiment is encoded as defined bythe well-known H.264/AVC standard. The encoded data stream includes notonly the image and audio data itself, but also the information to enablea decoder to decode the data stream, and information about the structureof the encoded data and the compression tools used during encoding.These streams may be referred to as “bitstreams,” but the units of theencoded data stream in various embodiments can be bits, bytes, picturefields, frames, or any other unit that is appropriate for theembodiment.

At step 335, the computing device 110 transmits the encoded data streamin real-time to the selected entry server using the assignment datareceived at step 325. The selected entry server acts as the entry pointfor the encoded data stream into the CDN 162, and receives the data inthe encoded data stream at step 340. The CDN 162 can then in turntransmit copies of the encoded data stream in real-time to the centraloffice proctoring center 180, and/or one or more geographicallydistributed proctoring centers 190, 192, 194, during the administrationof the examination. A proctor may then view the data stream via astreaming technique, and determine if any visual activity in theexamination environment constitutes activity not in accordance with theexam protocol.

FIG. 4 is a method for selecting an entry server to receive the encodeddata stream. At step 400, the assignment request is analyzed todetermine the geographic location of the computing device 110. In theillustrated embodiment, the geographic location of the computing device110 is determined by inspecting the IP address of the computing device110 which is included in the assignment request. In alternativeembodiments, other techniques for determining the geographic location ofthe computing device 110 may be used. For example, metadata in theassignment request may include the geographic location or addressprovided by the exam taker during registration or verification stages,or other metadata which can be used to identify the location of thecomputing device 110. In some embodiments, the geographic location oraddress of the computing device 110 may be determined utilizing theaddress information provided by the corresponding exam taker, which mayfor example be stored in the database server 174. As described above,this address information may be provided by the exam taker during theregistration or verification stages. Alternatively, the exam taker maybe prompted to provide such information prior to the launch of theexamination.

At step 405, a set of one or more entry servers of the entry servers160-1 to 160-6 are identified as possible entry points for the encodeddata stream into the CDN 162. These identified entry servers may be asubset of the entry servers 160-1 to 160-6, which may be selected basedon proximity to the determined geographic location of the computingdevice 110. For example, if it is determined that the computing device110 is located in San Francisco, Calif., the set of entry servers may bethose which are closest to San Francisco, Calif.

At step 410, expected performance metrics for different network pathsfrom the computing device 110 to each of the identified entry serversare calculated. For example, if the set includes entry servers 160-1,160-2 and 160-3, then the network paths from the computing device 110through the network 150 to each of the entry servers 160-1, 160-2 and160-3 are evaluated. These expected performance metrics may includebandwidth, packet loss, and delay for each of the various network paths.In some embodiments, these expected performance metrics are estimatedwithout requiring an analysis of the actual performance metrics of thenetwork paths. In such a case, relative values of the expectedperformance metrics may be determined by comparing the geographiclocation of the computing device 110 to the geographic and/or networklocations of the identified entry servers. For example, the network pathfor the geographically closest entry server may be assigned the lowestestimated delay value. Alternatively, the actual performance metrics ofthe networks may be collected and analyzed.

At step 415, the performance metrics for the set of identified entryservers are compared to one another. The selected entry server to act asthe entry point for the encoded data stream is then selected from theset based on this comparison. This selected entry server is selected asthe “optimum” entry point into the CDN 162 for the encoded data stream,to allow for an optimal first mile experience for the data stream. Thecriteria for the selection of the “optimum” entry server can vary fromembodiment to embodiment. The criteria may, for example, be performed byweighting each of the expected performance metrics for a given networkpath, and selecting the entry server based on comparison of the weightedresults. The criteria may, for example, be based on minimizing delayand/or packet loss of the uploaded data stream into the CDN 162. Otherand/or additional criteria such as upload bandwidth may also oralternatively be used.

The selection of the particular entry server may be based for example onwhich of the entry servers 160-1 to 160-6 is the closest to thecomputing device 110 in terms of network proximity. Network proximitymay for example be represented by the delay or number of handoffs forthe different network paths from the computing device 110 to each entryserver of the set. In such a case, the selected entry server maycorrespond to the network path with the least number of handoffs.

The selection of the entry server may alternatively be based for exampleon which of the entry servers 160-1 to 160-6 is closest to the computingdevice 110 in terms of physical proximity. In such a case, the selectedentry server may be the entry server which is geographically the closestto the computing device 110.

A backup entry server of the entry servers 160-1 to 160-6 may also beselected as an additional entry point for the data stream into the CDN162. The backup entry server may be selected for example as the secondmost “optimum” entry point into the CDN 162, using the same criteriaused to select the “optimum” entry point. In such a case, the assignmentdata transmitted to the computing device 110 by the application server172 will further request that the computing device 110 also transmit thedata stream to the selected backup entry server.

The utilization of the techniques for the optimization of the upload ofa data stream as described herein provide for high data stream qualitywith low startup latency, as well ensuring network friendliness andupload fairness. In addition, the techniques can be implemented withinexisting CDN infrastructures.

FIG. 5 illustrates a method 500 for transmitting an uploaded data streamof the examination environment to a proctor or other third party testassessment authority during an online proctored examination.

At step 505, the application server 172 receives a request to transmit aparticular data stream to a computing device during the administrationof the examination. In the illustrated example, the request for the datastream is from the computing device 191. As described above,alternatively the request may be a request that a data stream alreadybeing transmitted to the computing device 191 also be transmitted to adifferent computing device, so that proctoring responsibilities can beshared and/or shifted in real-time.

The request to transmit the data stream may for example be initiatedbased on user input provided to the viewing application 196 executing onthe computing device 191. Alternatively, the request may for example begenerated by the application server 172 on behalf of the computingdevice 191. For example, the request to begin transmitting the datastream to the computing device 191 may be automatically initiated by theapplication server 172 upon successful completion of the enrollment andauthentication of the exam taker.

In response to the request, at step 510 the application server 172selects an entry server in the entry servers 160-1 to 160-6 to transmitthe data stream in a streaming fashion to the computing device 191. Theapplication server 172 may select the entry server to transmit the datastream by analyzing the request in a manner similar to the techniquesdescribed above for selecting an “optimum” entry server which is theentry point for the data stream. For example, this determination can bemade based on one or more estimated performance metrics for a networkpath from the selected entry server to the computing device 191. Theperformance metrics can be computed using an estimated physical locationof the computing device 191, as well as entry server information such asthe physical location of each of the entry servers 160-1 to 160-6 storedin the database server 174.

Various techniques may be used to determine the geographic location ofthe computing device 191. For example, the geographic location may bedetermined by inspecting the IP address of the computing device 191which is included in the request. As another example, metadata in therequest may include the geographic location or address provided by theproctor during login to begin a proctoring session, or other metadatawhich can be used to identify the location of the computing device 191.In some embodiments, the geographic location or address of the computingdevice 191 may be determined utilizing the address information providedby the proctor, which may for example be stored in the database server174. Alternatively, the proctor may be prompted to provide suchinformation prior to the beginning of a proctoring session.

At step 515, the data stream is transmitted by the selected entry serverof the CDN 172 to the computing device 191 associated with the proctor.The data stream can then be displayed by the viewing application 196, sothat the proctor can monitor the examination environment surrounding theexam taker in real-time to detect possible aberrant behavior.

During the course of an examination, the proctor associated with thecomputing device 191 may be responsible for the simultaneous observationof a number of data streams of the examination environments surroundingdifferent exam takers. Upon detection of possible aberrant behavior by aparticular exam taker, the proctor may provide user input to the viewingapplication 196 requesting that the data streams of the other examtakers also be transmitted to the computing devices of one or more otherproctors. The viewing application 196 transmits this request to theapplication server 172. The application server 172 then determines whichcomputing devices are available to receive the data streams of the otherexam takers, and selects one or more appropriate entry servers totransmit the data streams of the other exam takers. In doing so, theproctor can more closely analyze the possible aberrant behavior by theparticular exam taker, without allowing the other exam takers to gounobserved. This selection of the entry servers to transmit the datastreams of the other exam takers can be similar to the techniquesdescribed above for selecting an “optimum” entry server to transmit thedata stream to the computing device 191.

The proctor may also request that the data stream of a test takerexhibiting possible aberrant behavior be transmitted to the computingdevice of another proctor or third party test assessment authority(e.g., a professor) for the purpose of validating or making anaffirmative determination that the exam taker in fact violated an examtaking protocol. Following a determination that an exam taker violatedan exam taking protocol, the proctor or other third party testassessment authority can then make a final or early decision regardingdisciplinary action.

Upon detection of aberrant behavior by an exam taker, the proctor mayintervene in the testing routine to resolve the aberrant behavior. Theproctor may intervene by providing user input to the viewing application196 which then transmits an indication to the application server 172that aberrant behavior has been detected. In response to the indication,the application server 172 operates in conjunction with secure testingapplication 140 to resolve the aberrant behavior. For example, theapplication server 172 may transmit a request to the computing device110 that the testing routine for the examination is no longer allowed tocontinue. Alternatively, the examination may be allowed to continue, butunder a flag of caution requiring further analysis of the data streamduring grading of the examination.

FIG. 6 illustrates an interface 600 that can be utilized in proctoringan online examination as might be observed on a computing device at thecentral office proctoring center 180 or the proctoring centers 190, 192,194. Interface 600 may allow for simultaneous observation of a number ofdata streams of the examination environment surrounding different examtakers. As shown in FIG. 6, the data stream of a single examinationenvironment 610 is being observed from a total of twelve availableexamination environments 620. As illustrated in FIG. 6, the examinationenvironment 610 being monitored exhibits aberrant behavior as reflectedby alert 630. Aberrant behavior may be automatically detected, orgenerated in response to proctor input. FIG. 6 also illustrates asession ID 640, which is unique to the exam taker; a proctoridentification 650, which identifies a proctor responsible for observingthe examination session; as well as a start and end time 660 for theexamination session. All of this information may be utilized ingenerating assessment data or logs following completion of theexamination. In some instances, aberrant behavior may result in the datastream of the examination environment being ‘exploded’ into a largerview (like in FIG. 7) in the case where the proctor is responsible formonitoring a large number of exam takers.

Upon the exhibition of aberrant behavior as reflected by alert 630 inFIG. 6, the specific examination session may be singled out for furtherinvestigation through the interface 700 illustrated in FIG. 7.

FIG. 7 illustrates a second interface 700 that can be utilized inproctoring an online examination. The interface 700 may be launched inresponse to detecting aberrant behavior observed in the interface 600 ofFIG. 6. The interface 700 of FIG. 7 (like that of interface 600 of FIG.6) displays real-time video of the examination environment 610 of theexam taker. Recording of the video may take place upon detection ofaberrant behavior for the purpose of validating or providing requisiteevidence related to addressing disciplinary activity following anaffirmative determination that an exam taker violated an exam takingprotocol. In some instances the aberrant behavior may simply be that theexamination environment needs to be modified to ensure properproctoring, which could include raising the light level or decreasingbackground noise (e.g., closing a window). A proctor may provide thisinformation to an exam taker.

The interface 700 of FIG. 7 also illustrates a current alert log 720that identifies the specific aberrant behavior that led to the automatedalert 630 in the interface 600 of FIG. 6. The proctor may log theoutcome of their determination related to the aberrant behavior inresponse log 730. Response log 730 allows a proctor to identify theparticular behavior that was at issue (e.g., an audio problem ormultiple people being present) (732) and the results of monitoring theaberrant behavior (734), which could include clearing the alert as afalse alert, terminating the examination, or allowing the exam tocontinue. A proctor may also launch an on-demand verification of audio,visual, or keystroke analytics. Notes related to the incident may alsobe maintained in notes section 736 to further detail the specificincident. In some instances, the proctor may launch a live chat sessionwith the exam taker while maintaining real-time observation.

The interface 700 may also maintain additional information such as ahistorical alert log 740 that maintains a running list of all aberrantbehavior for the exam taker in question as well as security information750, session information 760, and testing program information 770.Security information 750 may display specific information about an examtaker, including biometric information such as a photograph. Sessioninformation 760 may display information such as the name of the examtaker, the number of exam items answered, the number of breaks taken,and so forth as illustrated in FIG. 7. Information concerning specificprotocols related to the examination may be identified in exam programinformation window 770.

Logging of aberrant behavior may be tied to audio and video feeds of theexamination environment. In such instances, a proctor may simply log theunusual behavior but leave it to the exam assessment authority as to theultimate disciplinary behavior. Providing audio and video context tiedto the alert may be useful in this regard.

Computer-readable storage media refer to any non-transitory storagemedium and that may allow for instructions to be provided to a centralprocessing unit (CPU) for execution. Such media can take many forms,including, but not limited to, non-volatile and volatile media such asoptical or magnetic disks and dynamic memory, respectively. Common formscomputer-readable media include, for example, a floppy disk, a flexibledisk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROMdisk, digital video disk (DVD), any other optical medium, RAM, PROM,EPROM, a FLASHEPROM, and any other memory chip or cartridge.

Computer-readable storage medium, which are non-transitory in nature,differ from the various forms of transmission media may be involved incarrying one or more sequences of one or more instructions to a CPU forexecution. A bus carries the data to system RAM, from which a CPUretrieves and executes the instructions. The instructions received bysystem RAM can optionally be stored on a fixed disk either before orafter execution by a CPU. Various forms of storage may likewise beimplemented as well as the necessary network interfaces and networktopologies to implement the same.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. The descriptions are not intended to limit the scope of theinvention to the particular forms set forth herein. To the contrary, thepresent descriptions are intended to cover such alternatives,modifications, and equivalents as may be included within the spirit andscope of the invention as defined by the appended claims.

What is claimed is:
 1. A computer-implemented method for managingdistribution of a data stream from a distributed computing devicenetwork, the method comprising: receiving a data stream into thedistributed computing device network, the data stream corresponding toan environment surrounding a user of a first remote computing deviceduring execution of a testing routine for an examination; receiving arequest to transmit the data stream to a second remote computing device,the request identifying a geolocation of the second remote computingdevice and being received over the network during execution of thetesting routine; selecting a computing device of the distributedcomputing device network to process the request by transmitting the datastream to the second remote computing device, the computing deviceselected based on the geolocation of the second remote computing device;and transmitting the data stream from the selected computing device inthe distributed computing device network to the second remote computingdevice in response to the request.
 2. The computer-implemented method ofclaim 1, further comprising: receiving a second request to transmit thedata stream to a third computing device, the second request receivedduring execution of the testing routine; and transmitting the datastream from the second remote computing device in the distributedcomputing device network to the third computing device during executionof the testing routine in response to the second request.
 3. Thecomputer-implemented method of claim 2, wherein the second request isreceived from the second remote computing device.
 4. Thecomputer-implemented method of claim 1, wherein the data stream is avideo data stream of the environment surrounding the user of the firstremote computing device during the testing routine.
 5. Thecomputer-implemented method of claim 4, further comprising transmittingthe data stream to the second remote computing device includestransmitting the video data stream for display on the second remotecomputing device during the testing routine.
 6. The computer-implementedmethod of claim 1, further comprising receiving an indication from thesecond remote computing device that aberrant behavior in the environmenthas been detected through analysis of the data stream.
 7. Thecomputer-implemented method of claim 6, further comprising transmittinga request to the first remote computing device that the testing routinebe terminated.
 8. A non-transitory computer-readable storage mediumhaving a program embodied thereon, the program being executable by aprocessor to perform a method for managing distribution of a data streamfrom a distributed computing device network, the method comprising:receiving a data stream into the distributed computing device network,the data stream corresponding to an environment surrounding a user of afirst remote computing device during execution of a testing routine foran examination; receiving a request to transmit the data stream to asecond remote computing device, the request identifying a geolocation ofthe second remote computing device and being received over the networkduring execution of the testing routine; selecting a computing device ofthe distributed computing device network to process the request bytransmitting the data stream to the second remote computing device, thecomputing device selected based on the geolocation of the second remotecomputing device; and transmitting the data stream from the selectedcomputing device in the distributed computing device network to thesecond remote computing device in response to the request.
 9. Thenon-transitory computer-readable storage medium of claim 8, wherein themethod further comprises: receiving a second request to transmit thedata stream to a third computing device, the second request receivedduring execution of the testing routine; and transmitting the datastream from the second remote computing device in the distributedcomputing device network to the third computing device during executionof the testing routine in response to the second request.
 10. Thenon-transitory computer-readable storage medium of claim 8, wherein thesecond request is received from the second remote computing device. 11.The non-transitory computer-readable storage medium of claim 8, whereinthe data stream is a video data stream of the environment surroundingthe user of the first remote computing device during the testingroutine.
 12. The non-transitory computer-readable storage medium ofclaim 8, further comprising transmitting the data stream to the secondremote computing device includes transmitting the video data stream fordisplay on the second remote computing device during the testingroutine.
 13. The non-transitory computer-readable storage medium ofclaim 8, further comprising receiving an indication from the secondremote computing device that aberrant behavior in the environment hasbeen detected through analysis of the data stream.
 14. Thenon-transitory computer-readable storage medium of claim 8, furthercomprising transmitting a request to the first remote computing devicethat the testing routine be terminated.
 15. A system for managingdistribution of a data stream from a distributed computer network, thesystem comprising: a plurality of computing devices forming adistributed computer network, wherein each of the plurality of computingdevices includes a processor, memory, and a network interface; and aselected computing device of the plurality of computing devices that:receives at the network interface a data stream into the distributedcomputing device network, the data stream corresponding to anenvironment surrounding a user of a first remote computing device duringexecution of a testing routine for an examination, receives at thenetwork interface a request to transmit the data stream to a secondremote computing device, the request identifying a geolocation of thesecond remote computing device and being received over the networkduring execution of the testing routine, and executes instructionsstored in the memory, wherein execution of the instructions by aprocessor of the selected computing device: selects a computing deviceof the distributed computing device network to process the request bytransmitting the data stream to the second remote computing device, thecomputing device selected based on the geolocation of the second remotecomputing device, and transmits the data stream from the selectedcomputing device in the distributed computing device network to thesecond remote computing device in response to the request.
 16. Thesystem of claim 15, wherein the selected computing device furtherreceives a second request to transmit the data stream to a thirdcomputing device, the second request received during execution of thetesting routine and transmits the data stream from the second remotecomputing device in the distributed computing device network to thethird computing device during execution of the testing routine inresponse to the second request.
 17. The system of claim 16, wherein thesecond request is received from the second remote computing device. 18.The system of claim 15, wherein the data stream is a video data streamof the environment surrounding the user of the first remote computingdevice during the testing routine.
 19. The system of claim 15, whereinthe selected computing device further receives an indication from thesecond remote computing device that aberrant behavior in the environmenthas been detected through analysis of the data stream.
 20. The system ofclaim 19, wherein further execution of the instructions transmits arequest to the first remote computing device that the testing routine beterminated.